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FOREWORD 



The Carl D. Perkins Vocational and Applied Technology 
Education Act Amendments of 1990 included a recjuirement that states 
develop a system of performance measures and standards for use in 
program evaluation and improvement. This type of requirement had 
previously been included in the Job Training Partnership Act. 
However, this was the first time it had been included in vocational 
education legislation. 

Implementing a system of performance measures and standards is 
a large undertaking. Since this was the first time it had been 
included in vocational education, information is needed regarding 
the implementation process . This information could be used for 
improving vocational education programs and guiding future policy 
initiatives. As of December 1, 1992, no information was available 
concerning the actual systems of performance measures and standards 
that had been approved by each state. 

The project staff is indebted to the State Directors of 
Vocational Education and their staff, who provided the information 
on the performance standards and measures systems. This study was 
sponsored by the Graduate School at The Ohio State University 
through a seed grant for the Comprehensive Vocational Education 
Program. Special appreciation is extended to Dr. Roy A. 
Koenigsknecht, Dean of the Graduate School for providing this 
funding. Additionally, the researchers are grateful to the 
following members of the Coordinating Council for the Comprehensive 
Vocational Education Program: Drs. Aaron J. Miller, Sharon V. 
Redick, and Michael Scott. The Comprehensive Vocational Education 
Program is an interdepartmental effort of three colleges. The 
colleges and the Deans include: Agriculture, Dr. Bobby D. Moser; 
Education, Dr. Nancy L. Zimpher; and Human Ecology, Dr. Jerelyn 
Schultz. Ms. Connie Myers provided secretarial services to the 
project. 

R. Kirby Barrick 
Professor and Chair 
Coordinating Council for 

Comprehensive Vocational Education 
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INTRODUCTION 



In the last decade there has been an increasing level of 
dissatisfaction with the quality of public education in the United 
States. The initial sound of alarm regarding the inadequacies of 
our education system began with the release of A Nation at Risk 
(National Commission on Excellence in Education, 1983) . Since that 
time a number of other National reports have signaled similar 
concerns. Among these studies were: The Forgotten Half: Non 
College Youth in America (The William T. Grant Foundation 
Commission on Work, Family and Citizenship, 1988) ; America * s 
Choice: High Skills or Low Wages! (Commission on Skills of the 
American Workforce, 1990) ; America 2QQ0: An Education Strategy (U. 
S. Department of Education, 1991); What Work Requires of Schools; 
A SCANS Report for America 2000 (Secretary's Commission on 
Achieving Necessary Skills, 1991) ; Education Counts (Special Study 
Panel on Education Indicators, 1991) ; and Learning a Living; A 
Blueprint for High Performance (Secretary's Commission on Achieving 
Necessary Skills, 1992). Each of these studies called for major 
changes in the U.S. educational system by stressing the gap between 
the demands of the future and the present level of preparadness of 
America's youth to meet these requirements. 

Accompanying the need for improved educational programs was a 
call for better evidence regarding the accountability of public 
education. Hill and Bonan (1991) defined accountability in the 
following manner: 

Accountability is a relationship between two persons in 
which four conditions apply; first, one person expects 
the other to perform a service or accomplish a goal; 
second, the person perfomming the activity accepts the 
legitimacy of the other's expectation; third, the person 
performing the activity derives some benefits from the 
relationship; and fourth, the person for whom the 
activity is performed has some capacity to affect the 
other's benefits, (p. 35) 

The acceptance of the concept of accountability in education has 
been widespread. In fact, as White (1990) explained; 

Accountability has become a guiding principle in the way 
states have approached new ways to improve education. 
Thus, systems to measure accountability have become a 
major ^business' in The United States (Odden, 1990) . 
Schools are expected to act like businesses, and account 
for their successes and failures; and though it has never 
been easy to apply quantitative measures to complex 
educational processes and outcomes, more and more school 
systems have been doing just that. (p. 1) 



Accountability systems in education have traditionally relied 
on reviews of inputs and processes of the educational systems • 
However, the focus on accountability systems in education has been 
changing to one of assessing outcomes (McCaslin, 1990) • The 
National Governors' Association has led a movement to reorient 
accountability to include outcomes (Henry, McTaggert, and McMillen, 
1992)* "Education 2000", the current statement of the national 
goals for U^S . education, also reflected an emphasis on 
accountability. 

Accountability efforts can offer important information for 
improving the educational system* Apling (1989) stated: 

To improve the education of America's children, 
educational programs at every level must set standards of 
excellence and hold students, teachers, and schools 
accountable for meeting those standards* (p« 4) 

The movement toward more emphasis on accountability and 
evaluation has had an impact on vocational education. At the time 
of the hearings to reauthorize the Carl D. Perkins Vocational 
Education Act, the Office cf Technology Assessment (1989) reported 
that, "there is now widespread consensus for including the 
vocational education system in the national debate over school 
reform and academic excellence" (p. 2). Later, Jennings (1991) 
remarked : 

The newly reauthorized Carl D. Perkins Vocational and 
Applied Technology Act is a call from Congress to 
vocational educators. The message: It's time to get off 
the sidelines and get into the game. With dozens of 
reports indicating that a decade of state and local 
reform efforts have failed to improve U.S. scudents' 
performance, vocational education is being asked to play 
a leadership role. (p. 18) 

Vocational education has considered accountability and 
evaluation essential activities for many years. The next section 
highlights these efforts. 

VOCATIONAL EDUCATION'S INVOLVEMENT WITH ACCOUNTABILITY 

AND EVALUATION 

The 1963 Vocational Education Act was one of the first 
examples of federal legislation requiring states to conduct 
evaluations of their programs. In response to this mandate, most 
vocational education programs conducted follow-up studies to 
determine the degree to which vocational education graduates found 
placements in jobs that were related to their training to meet this 
requirement. Additionally, states often used program reviews 
conducted by local education personnel and external evaluators to 
chart their future directions. 



The 1968 vocational Education Amendments continued to 
emphasize state evaluation activities • It expanded the concept by 
giving evaluation responsibilities to state advisory councils. 

The Education Amendments of 1976 (P.L. 94-482) further 
expanded the responsibility for evaluation and had at least 28 
references to different forms of evaluation (Wentling, 1980). 
These evaluation requirements were designed to increase the 
responsiveness of vocational education to changing labor markets 
and required states to evaluate programs every five years and to 
determine the extent to which programs completers and leavers: (1) 
found employment in occupations related to their training, and (2) 
were considered by their employers to be well-trained and prepared 
for employment. 

Seven different groups were specifically charged with conducting 
vocational education evaluations in the 1976 Amendments: (1) the 
state boards of vocational education, (2) the state advisory 
councils for vocational education, (3) the National Advisory 
Council for Vocational Education, (4) the U.S. Office of 
Education's Bureau of Occupational and Adult Education, (5) the 
National Institute of Education, (6) The National Center for 
Education Statistics, and (7) the U.S. Office of Education's Office 
of Evaluation and Dissemination. 

The Carl D. Perkins Vocational Education Act (P.L. 98-524), 
enacted in 1984, charged the states to: 

develop measures for the effectiveness of vocational 
education programs including measurements such as — 

(i) the occupations to be trained for reflect a 
realistic assessment of the labor market of the 
state, 

(ii) the levels of skills to be achieved in 
par-^-icular occupations, which will reflect the 
hiring needs of employers; and 

(iii) the basic employment competencies to be 
used in performance outcomes, which will reflect 
the hiring needs of employers; (p, 2447) 

The Carl D. Perkins Vocational and Applied Technology 
Education Act of 1990 continued to include evaluation 
specifications. It required states to develop core standards and 
measures of performance for secondary and postsecondary vocational 
education programs. These performance measures were to include: 

(1) measures of learning and competency gains, including 
student progress in the achievement of basic and more advanced 
academic skills; 
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(2) 1 or more measures of performance, which shall include 
only — 

(A) competency attainment; 

(B) job or work skill attainment or enhancement including 
student progress in achieving occupational skills 
necessary to obtain employment in the field for which the 
student has been prepared, including occupational skills 
in the industry the student is preparing to enter; 

(C) retention in school or completion of secondary school 
or its equivalent; and 

(D) placement into additional training or education, 
military service, or employment; 

(3) incentives or adjustments that are — 

(A) designed to encourage service to targeted groups or 
special populations; and 

(b) for each student, consistent with the student • s 
individualized education program developed under section 
614(a)(5) of the Education of the Education of the 
Handicapped Act, where appropriate; and 

(4) procedures for using existing resources and methods 
developed in other programs receiving Federal assistance, 
(p. 770-771) 

The Carl D. Perkins Vocational and Applied Technology Act of 
199C was authorized during a time when "performance incentives and 
quality indicators were very much in vogue" (White 1990, p, 2). 
Congress had authorized the largest yearly outlay for vocational 
education ever (1.6 billion dollars) and wanted increased 

=^ccountability. Therefore, Congress mandated that states " set 

up extensive systems for evaluating programs and the effects of 
Perkins dollars" (Wilcox 1991, p. 16), 

Although the concern for quality and accountability was a 
major reason for establishing statewide systems of performance 
measures and standards, several studies and groups also had 
recommended the use of performance measures and standards. The 
Office of Technology Assessment (1989) reported: "The education 
and business communities now increasingly support the view that 
more exacting measurement of the quality of high school vocational 
programs may be needed" (p. 1) . The recommendation of The National 
Assessment of Vocational Education concerning accountability for 
vocational education was accepted by Congress when the 
reauthorization of Perkins was debated (Apling, 1989) . Another 
important influence, as cited by White (1990), for the adoption of 
performance measures and standards was the National Association of 
State Directors of Vocational Education. This association had 
supported the adoption of an accountability system consisting of a 
system of performance measures and standards for local vocational 
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programs. National reports concerning accountability had not only 
directed attention to the quality of educational programs but 
••...also raised many questions about how effectiveness is to be 
defined and attained** (Ewell 1988, p. 53). Apling (1989) stated 
the reason for using performance standards: 

Performance standards have been mandated for various 
social programs because of concern that these programs 
are not working as well as they could and that adopting 
principles of the marketplace will improve these 
programs, (p. 3) 

Although the Carl D Perkins Vocational and Applied Technology 
Education Act Amendments of 1990 presented a specific requirement 
for a system of performance measures and standards, this was not 
its first appearance on the national policy scene. The Job 
Training Partnership Act (JTPA) had previously required a system of 
performance measures and standards. 

THE JTPA INFLUENCE ON ACCOUNTABILITY AND EVALUATION 
EFFORTS IN VOCATIONAL EDUCATION 

Performance standards were first adopted as an instrument of 
national human resource policy in the employment and training area 
with the passage of the Job Training Partnership Act of 1982 (P.L. 
97-300) . The measures that were developed for these programs 
included: placement and retention in unsubsidized employment, 
earnings, and reductions in public assistance. Performance 
standards had made a major impact on how these programs were 
conducted. Butler (1988) had made the following assessment of the 
use of performance measures in JTPA: 

The most significant outcome-oriented practice has been 
the development of formal national, state, and locally- 
administered systems of outcomes measures, and standards 
for aggregated program achievement. This so called 
^performance standard* system-driven by a nationally- 
derived set of outcome numbers against which the 
performance of local administrative entities, called 
Service Delivery Areas, or SDAs, (and in turn, states) 
are measured-has become the basis for judgement about the 
effectiveness of local programs, and when they are 
aggregated, about the system as a whole. This 
rigorously-applied set of measures and standards has the 
effect of turning the system on its head to comply. The 
measures have recently been expanded, and there have been 
annual adjustments in the standards, but in general there 
has been great consistency over the several years of 
JTPA, providing a good basis of experience with outcome 
mea5;,ures for a national program, (p. 2) 
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For the federal government, priorities in implementing JTPA 
1982 were to hold local providers responsible for the outcomes that 
were attained, to encourage efficient service, to create incentives 
for effective management of local programs, and to foster 
acceptance of the program by business and industry (Dickinson & 
West 1988). Though the record for JTPA was positive — especially 
compared to the Comprehensive Employment and Training Act (CETA) , 
the federal training program that preceded JTPA — some concerns were 
raised • Among the concerns reported were : ( a ) inappropriate 
targeting of participants and services (Apling, 1989) , (b) 
questionable performance measures (Frazier, 1991) , (c) problems in 
defining seirvices, outcomes, etc* (Frazier), and (d) inconsistent 
and incomplete data (Office of Technology Assessment, 1989) • The 
influence of performance measures for program accountability can 
become so great, that it results in the unintended consequences 
such as reduced services to targeted groups • ( Dickinson & West 
1988) 

Some problems were reported when agencies received incentives 
and adjustments for serving special populations. Apling (1989) 
reported that in JTPA programs where adjustments were in use, data 
was difficult to obtain and verify concerning the special 
conditions that warranted adjustments. When asked to justify 
funding for programs on the basis of performance standards, JTPA 
representatives at times found it difficult to produce hard data to 
support their claims. Further, there were questions about the 
programs becoming so outcomes driven, that the mission of the 
program became secondary to producing adequate "numbers." Butler 
(1988) suggested that: 

There is an underlying policy question abet the 
application of performance standards under JTPA. That 
is, do perfonaance standards achieve the ^investment in 
human capital • agenda of the Act, or does the approach 
artificially force states and localities to opt for quick 
numbers which will yield only short-term results? 
Unfortunately there are no reliable data which will 
answer that most basic question. . .there is little 
systematic information on what services were actually 
provided and virtually none at all on who else might have 
been served had the programs been designed or supported 
dif ferently-or even on who applied for services but was 
rejected on grounds that they might not succeed. 
Instead, we must rely on very partial official data on a 
few aggregate measures, and on the opinion and self- 
reporting by professionals involved in the program, (p. 
7) 

Regardless of the problems associated with performance 
measures and standards, there are advantages to their use. Butler 
(1988) stated: 



JTPA has demonstrated that it is possible to establish a 
national program substantially driven by clearly- 
specified, measurable outcomes* Moreover, it is possible 
to tie those outcomes to funding, not only at the federal 
and state levels, but even down to the level of actual 
service deliver^/, As policies which underlie vocational 
education are developed, JTPA's experience should 
encourage planners that a focus on outcomes can be 
achieved* (p. 21). 

The experience in evaluating local JTPA programs was judged to 
be of sufficient value by Congress that a similar system of 
performance standards and measures were included in The Carl D. 
Perkins Vocational and Applied Technology Education Act of 1990. 
As Apling (1989) reported to Congress: 

The use of performance-based management principles in the 
public sector with an emphasis on program outcomes or the 
^bottom line* reflects in part a belief that outcome 
testing will improve the accountability, management, cost 
effectiveness, and ultimate performance of public 
programs, (p. 4) 

Since the passage of the Carl D. Perkins Vocational and 
Applied Technology Education Act of 1990, states have been revising 
and developing their systems of performance standards and measures. 
The next section presents information related to using performance 
measures and standards. 

USING PERFORKANCE MEASURES AND STANDARDS 

The impact of the 1990 Amendments will be felt in the amount 
of attention paid to the outcomes of programs. However, measures 
and standards must be carefully constructed so as not to hinder the 
process they seek to evaluate. States must make choices such as 
whether to emphasize the development of academic measures centered 
on benchmarking (meeting an outside accepted standard) or on value 
added. As Ewell (1988) stated: 

From a policy perspective, however, the issue can be real 
and concrete: Are institutions and programs to be judged 
primarily in terms of the degree to which they Mevelop 
talent* or in terms of the degree to which their ultimate 
products meet accepted standards? (p. 64) 

Some have questioned the merits of these systems. Apling 
(1989), for example, reported that: 

Unless carefully drawn and implemented, perfonrjince 
standards may not improve performance and might adversely 
influence performance; the diversity and complexity of 
vocational education in this country may make performance 
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standards even more difficult to develop and impleraent 
than they have been for other programs (p. 3) 

To accept measures of outcomes alone as an evaluation scheme 
would not prove useful. McCaslin (1990) indicated: 

if information on student achievement in vocational 

and academic skills is all that is available, how can 
teachers and administrators know what worked and did not 
work? Upon what basis should decisions to add and drop 
programs be made? What should be changed in order for 
vocational students to achieve at higher levels? (p. 7) 

White (1990) echoed this concern by citing Asche; 

Outcome indicators do not provide information on how or 
why such outcomes are produced or what to do if results 
are judged unsatisfactory. (p. 31) 

Others have expressed confidence in the use of a system of 
measures and standards. One example is the Oklahoma Department of 
Vocational and Technical Education (1992) who published the 
following: 

We believe that the requirements set out in Carl Perkins 
support the delivery of high quality vocational programs, 
(p. 2.3) 

When used within the context of a broader and more comprehensive 
system of evaluation, the use of perfomnance measures can be 
]::elpful. If needs and processes are evaluated in addition to 
outcomes (McCaslin, 1990) , a much more valuable picture of the 
status of the program can be determined. Systems of measures and 
standards can provide information for comparing outcomes across 
programs and states. When dealing with occupational training, 
there is an expectation in the workplace, that measures and 
standards can be used as benchmarks for performance. Measures and 
standards for vocational programs also can serve to provide the 
framework for assessing student progress against an accepted 
standard , and for identi f y ing programs where outcomes are low , 
suggesting inadequacies in the system. 

The use of systems of performance measures, though valued in 
the JTPA model, will undoubtedly bring a host of problems for 
vocational education. These problems include the questionable 
validity of information gathered on program participants (Office of 
Technology Assessment 1989) , especially if the information is self 
reported; the difficulty in assessing the effects of a program in 
the development of academic skills (Office of Technology 
Assessment) ; the challenges in meeting a mixed set of standards 
(Apling, 1989) ; insufficient investment to insure high quality for 
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all the measures (White, 1990) ; and difficulty in achieving program 
improv<*.ment when evaluation focuses on outcomes (McCaslin, 1992) . 

It could well be that vocational educators, feeling similar 
pressure to perform as did JTPA professionals, could develop 
practices of screening potential students, collecting inadequate 
data, and even altering the types of courses offered in an attempt 
to make the data look good. Will it happen? Only thorough 
investigation over time will reveal the answer to that question. 

PROBLEM STATEMENT 

The move to implement a system of performance measures and 
standards in vocational education is a large undertaking. The 
states had until September 25, 1992 to implement the systems of 
standards and measures (Federal Register, August 14, 1992) . Many 
of the states found it necessary to develop new evaluation 
procedures as a result of the mandates of the Carl D. Perkins 
Vocational and Applied Technology Education Act of 1990. Yet, 
relatively little information is available regarding this process. 
Hoachlander and Rahn (1992) gathered information in 1991 from the 
states in an effort to deteraine the expected makeup of the 
systems. However, as stated by Hoachlander and Rahn, "The systems 
actually implemented in fall 1992 may look substantially different, 
as states continue to develop performance measures and standards." 
(p. 2) By December 1, 1992, no information was available 
concerning the actual systems of core standards and performance 
measures that have been adopted by each state. This infoinaation 
would be useful in further developing and improving the system of 
performance measures and standards and in meeting the requirements 
of the Carl D, Perkins Vocational and Applied Technology Education 
Act of 1990. 

PURPOSE AND OBJECTIVES 

This study was sponsored by the Graduate School at The Ohio 
State University through a seed grant for the Comprehensive 
Vocational Education Program. The purpose of this study was to 
examine the system of performance measures and standards that had 
been approved in each state of the United States in response to the 
Carl D. Perkins Vocational and Applied Technology Act of 1990. The 
specific objectives were: 

1. To determine what types of performance measures had 
been approved in each state. 

2. To identify how states have decided to assess their 
performance. 



METHODOLOGY 



Descriptive-survey and content analysis research methods wera 
used in this study. The names and addresses of the 54 state 
directors of vocational education were used to generate the 
population for this investigation. For the purposes of this study, 
a state was defined as including all 50 states in the United States 
and the District of Columbia, Puerto Rico, Guam, and The Virgin 
Islands. 

An initial letter (see Appendix A) was sent to each state 
director on November 24, 1992 requesting documents describing the 
system of performance measures- and standards that had been approved 
by their state board for vocational education. This strategy was 
used to minimize the amount of time and energy that would be 
required to provide the information. Approximately four weeks 
later, a f ollowup letter, containing the original request, was 
again sent to the 19 state directors thrt had not responded. A 
third and final follow-up letter was sent on February 1, 1993 to 
the remaining 7 state directors that had not responded. On March 
3, 1993 phone calls were made to the remaining 5 state directors 
from which no response had been received. As of April 15, 1992, 52 
of the 54 states (96%) had responded with information concerning 
their approved systems of measures for secondary vocational 
education programs and 50 (93%) had responded with information 
concerning their approved systems of measures for postsecondary 
programs. Two states (Iowa and The Virgin Islands) indicated that 
their system of measures and standards had not been approved by 
their state boards. Additionally, Georgia and Arkansas did not 
report their system of postsecondary performance measures and 
standards. 

Once the documents had been reviewed, a content analysis was 
conducted. The analysis resulted in a listing, by states, of the 
measures that the states had adopted, using the categories listed 
in section 115 of the Carl D. Perkins Vocational and Applied 
Technology Education Act of 1990. The following categories were 
established: (a) basic academic skill, (b) advanced academic skill, 
(c) competency attainment, (d) workskill attainment, (e) 
retention/completion, (f) placement, (g) service to special 
populations, and (h) other measures. 

A summary sheet was developed for each state, categorizing the 
data on adopted measures. On March 12, 1993, the sximmary sheets 
were mailed to the state directors of vocational education (a copy 
of the summary sheet, instructions and the cover letter are 
contained in Appendix B) . Personnel from each state were asked 
review, verify and amend the listing as necessary. When 
discrepancies occurred, a further review of the documents was 
carried out. If necessary, a followup call was made to the state 
director of vocational education • s office for additional 
clarification. 
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FINDINGS 



This section reports the measures that states have had 
approved in implementing their statewide system of core standards 
and measures of performance. First, the measures approved for 
secondary vocational education will be presented. These will be 
followed by the measures approved for postsecondary vocational 
education. 

Approved Measures for Secondary Vocational Education 

The vast majority of the measures reported by the states have 
been approved for implementation in 1993, However, several 
measures were reported that were approved for implementation in 
1994 or later. As indicated previously, Iowa and the Virgin 
Islands had not yet approved standards for their system. 
Therefore, information from these areas was not included i'l this 
report. The average number of performance measures approved for 
secondary vocational education was 10. 

Academic Skills 

The information presented in Table 1 identifies the areas for 
which states have approved measures of learning and competency 
gains in academic skills. These academic skills include two types: 
basic and advanced. For purposes of this report the academic 
skills were classified as either reading, language, mathematics, 
science or "other". Thirty states were using the same set of 
measures for both basic and advanced academic skills. The 
remaining twenty two states were using different sets of measures 
for basic and advanced academic skills. At the secondary 
vocational education level, two of the reporting states did not 
indicate that they had approved any basic or advanced academic 
skill measures. 

Basic Academic Skills . The area that states most often had 
approved as a measure of basic academic skills (see Table 1) was 
mathematics (85%) . This was followed by reading (80%) , language 
(77%) and science (35%). A total of 44% of the states indicated 
that they had approved an "other" measure of basic academic skills 
for their system of standards and measures of performance. 
Examples of "other" basic academic skills included measures in 
areas such as social studies, grade point averages, critical 
thinking, and problem solving. 

Advanced Academic Skills > The areas in advanced academic 
skills (see Table 1) that were reported approved by each state 
followed a pattern similar to that reported for basic academic 
skills. More than three-fourths of the states reported a 
mathematics measure for advanced academic skills (76%) . About two- 
thirds (65%) of the states reported using reading and language 
measures. Less than half of the states reported using measures in 
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the area of science (^1%) of the states. In the area of "other" 
advanced academic sklxls, 43% of the states had approved measures 
such as critical thinking, problem solving, and social studies. 
Six of the states reporting did not include any approved measures 
of advanced academic skills. 

Other Measures of Performance 

Table 2 presents information on other measures of performance 
that have been approved by the states. Only two states had not yet 
approved any measures of performance for secondary vocational 
education programs. These areas are discussed below and include: 
competency attainment, work skill attainment, program completion, 
high school graduation, placement, percent served, and gender mix. 

Competency Attainment . Competency attainment was generally 
defined by the states as basic employability skills. Approximately 
one-half of the states (44%) reported using competency attainment 
performance measures. 

Work Skill Attainment . This area tended to be defined by the 
states as including measures of specific occupational skills 
attainment. About three-fourths of the states (72%) reported that 
their states had approved performance measures on work skill 
attainment. 

Program Completion . States generally referred to program 
completion as the rate at which students fulfilled the requirements 
of their program. Almost one-half (46%) of the states reported 
having this type of performance measure approved. 

High School Graduation . Higl\ school graduation referred to 
the rate of students who had successfully completed the 
requirements for graduation in their school or its equivalent 
(e.g.. General Equivalency Diploma). One-half (50%) of the states 
reported using this as a performance measure. 

Placement . This measure of program effectiveness has been 
used for some time in vocational education. Related placement 
refers to individuals who have obtained employment in an area 
closely related to their area of education and training. Any 
placement refers to obtaining any type of job after completing 
their program of studies. Approximately one-half (46%) of the 
states reporting using related placement as a performance measure. 
Whereas, sixty one percent of the states reported using any 
placement as a performance measure. Nine states reported using 
both types of placement as performance measures. A total of 92% of 
the states reported using some type of placement measure. 

Percent Served . This performance measure referred to the 
percent of the high school aged special population students that 
were enrolled in vocational education programs. Slightly more than 

14 



one-half (52%) of the states reported that they were using this 
type of performance measure. 

Gender Mix , Gender mix referred to the percentage of male and 
female students who were enrolled in vocational education programs. 
Approximately one-third (31%) of the states indicated that this 
type of measure was being used in their system. 



15 



eg 



U 

c, 

0) 





c 


c 


.< 


0) 








0) 




0 


0) 


«3 










rH 




0) 







04 (0 

o ^ 



1 



X X 



XXX 



X X 



X X 



X X 



X 



X XXX XX 



XXX 



X XX XX 



^1 X X X 



X X 



X X 



X X 



X 



X 



X X XX 



X XX 



X X X X 



X X 



X 



XXX 



X X 



X X X X 



X X 



X 



X X X X X 



X 



X X X X X 



XX XX 



X 



X X 



X X 



XXX 



X 



X X 



X X 



X X 



►a « M 



O Eh W < 
U U Q O 



H Q 55 •< 
K H H H H 



CO W Q 

h3 



X X 



X X 



X X 



X 



X 



X X 



CO O w 

XX x^ 



16 



1 



0) 

c 

•H 
C 

o 
o 

eg 

0) 
•H 

EH 



0) 



0) 0) 
0) 0) 

04 CO 





>^ 




c 


c 


.< 


0) 




§ 








0 


0) 




+J 


iH 






iH 




0) 







o (d 
CO :3 

CP M 

•H a 
a 

c 
o 

M 0) 

o a 
M g 
cu o 
o 



CO 
M 

u 
o 
IS 



>1-p 
o c 



c 

0) 
0) 



O 4J 



0) 
CO 



X X 



XXX 



XX XX 



XXX 



X X 



XX X 



XXX XX X 



X X 



XX XXX 



XX X X X X X 



XXX 



X X 



X X 



X loo 

UN in 



X X 



X X X X 



X 



X |r- o 
oj in 



X X X X 



X XX 



XX [in vo 

CM ^ 



X X X X 



X X X X 



XX XX xxxxx 



XXX 



X 



C\ eg 



X X 



XXX 



O Q ffi « fiJ 
jz: jz: o o o 



i< H O Q 55 
0< CO CO H 

17 



X Eh 



> 3: 



a s oi H 

Q O 0< > EH o< 



eg ^ 



O 



"ERIC 



A pproved Measures for Postsecondairv Vocational Education 



Again, most of the measures reported by the states were 
approved for use in 1993. However, some of the measures were 
approved for use after 1993. Iowa, and the Virgin Islands reported 
that they had not yet approved standards for postsecondary 
vocational education programs. Information concerning 

postsecondary measures for Georgia and Arkansas was unavailable. 
Therefore, information from these areas was not included in this 
report. The average nximber of performance measures approved for 
postsecondary vocational education was 8. A total of four states 
did not report having any basic or applied academic skill measures 
approved for their postsecondary programs. 

Academic Skills 

The information presented in Table 3 identifies the ar^as for 
which states have approved measures of learning and competency 
gains in academic skills for postsecondary vocational education 
programs. These academic skills include two types: basic and 
advanced. For purposes of this report the academic skills were 
classified as either reading, language, mathematics, science or 
"other". Nineteen states were using the same set of measures for 
both basic and advanced academic skills. The remaining thirty-five 
states were using different sets of measures for basic and advanced 
academic skills. 

Basic Academic Skills . The area that states most often had 
approved as a measure of basic academic skills (see Table 3) was 
math (56%). This was followed by language (54%), and reading 
(48%). Science was reported being used as measures by 15%. A 
total of 48% of the states indicated that they had approved an 
"other" measure of basic academic skills for their system of 
standards and measures of performance. Examples of "other" basic 
academic skills for postsecondary vocational education included the 
following: course completion, GPA, social studies, and thinking 
skills. One state did not report any approved basic academic skill 
measures . 

Advanced Academic Skills . Table 3 also reports the areas in 
advanced academic skills that were reported approved for 
postsecondary vocational education in each state. In the advanced 
academic skills area, 48% of the states reported they were using 
performance measures related to mathematics and 44% reported using 
r asures related to language. Approximately one-third (30%) of the 
states were using measures in the reading area. Only eight states 
(15%) were using performance measures related to science. 
Approximately 40% of the states reported using performance measures 
related to other advanced academic skill areas. This "other" 
category included measures such as: problem solving, higher order 
thinking, and interpersonal relations. Three states did not report 
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any approved advanced academic skill measures for their 
postsecondary vocational education programs* 

Other Measures of Performance 

The information in Table 4 presents information on other 
measures of performance that had been approved by the states for 
their postsecondary vocational education programs. At the 
postsecondary level, six states had not yet had any measures of 
performance approved. The other measures of performance were 
similar to those presented for secondary vocational education and 
included: competency attainment, work skill attainment, program 
completion , placement , percent served , and gender mix . These 
performance measures will be discussed below. 

Competency Attainment . States tended to define competency 
attainment as the development of employability skills. Only one- 
third (33%) of the states reported that they had approved 
competency attainment performance measures for postsecondary 
vocational education programs. 

Work Skill Attainment . This area was generally defined by the 
states as including measures of the extent to which students had 
developed specific occupational skills. Approximately two-thirds 
of the states (63%) reported that they had approved this type of a 
performance measure. 

Program Completion . States tended to refer to program 
completion as a measure of the ratio of students who initially 
enrolled to those who met the requirements/ outcomes of the program. 
Nearly three-fourths (70%) of the states reported having approved 
this type of performance measure. 

Placement . Two types of placement rates were reported by the 
states: related placement and any placement. Related placement 
refers to individuals who have obtained employment in an area 
closely related to their area of education and training. Any 
placement refers to obtaining any type of job after completing 
their program of studies. Approximately one-half (48%) of the 
states reporting using related placement as a performance measure. 
Fifty six percent of the states reported using any placement as a 
performance measure. Ten states reported using both types of 
placement as performance measures. A total of 88% of the states 
reported using either related or any type of placement as an 
approved performance measure. 

Percent Served . This performance measure referred to the 
percent of the special population students that were enrolled in 
postsecondary vocational education programs. Slightly more than 
one-half (56%) of the states reported that they were using this 
type of performance measure. 
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Gender Mix . Gender mix referred to the percentage of male and 
female students who were enrolled in these postsecondary vocational 
education programs. Slightly less than one-third (31%) of the 
states indicated that their system was using this type of measure. 

T ypes of Assessment Techniques for Academic and Other Performance 

States reported using a wide variety of types of assessment 
instruments, procedures and processes for measuring academic and 
other types of performance. Four general categories were 
identified for academic performance, these included: state 
developed assessment techniques, locally developed or selected 
techniques, nationally recognized measures, and other assessment 
techniques. Five general categories of assessment techniques were 
identified for use with other performance. Types of measures used 
for other performance included: state developed, locally developed 
or selected, nationally recognized, occupational certification or 
licensure, and other assessment techniques. 

Locally selected or developed assessment techniq^ies included all 
those left to the discretion of the local education agency. In 
some states, the local agency develops the measuring instrument and 
submits it for state approval. In other states, the state 
publishes a list of suggested assessment instruments from which the 
local agency may select. In still other states, the local may 
choose from any available nationally recognized, state developed or 
locally developed assessment instrument with no state approval. 

Nationally recognized instruments include standardized tests of 
academic or occupational performance. These may also include 
nationally recognized checklists and competency guides. 

Occupational certification or licensure includes assessment 
procedures adopted by national or state occupational boards for 
certifying competency in specific occupations. Examples include 
nursing boards, cosmetology boards, and automotive certification 
agencies. 

The other category includes instruments to measure student 
performance in coursework. These include successful completion of 
courses, grades, and the use of student portfolios. 

The infor/aation presented in Table 5 identifies the number of 
states using the various assessment techniques used in measuring 
basic and advanced academic performance. The most frequently used 
techniques for assessing academic performance for secondary 
vocational education were state developed. More than one half of 
the states reporting the use of measures of reading, language and 
math used state developed assessment techniques, and about three- 
fourths of the states reporting the use of a science measure were 
using state developed techniques. 
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The most frequently used assessment techniques for assessing 
academic performance for postsecondary programs were locally 
selected or developed. About fifty percent of the states reporting 
using reading, language, and math as measures of academic 
performance were using locally developed or selected assessment 
techniques. Almost 90% of those reporting the use of a measure of 
science used locally developed measures for assessing performance. 

Table 6 summarizes the assessment techniques used in measuring 
competency attainment and workskill attainment. The most popular 
method for assessing competency attainment in both secondary and 
postsecondary programs reported by the states were locally selected 
and developed assessment techniques. Approximately two-thirds of 
the states reporting the use of a competency attainment measure for 
both secondary and postsecondary programs were using locally 
selected or developed methods. 

Most frequently used among the methods for assessing workskill 
attainment were locally selected or developed techniques. Among 
states reporting a measure of workskill attainment, about 70% were 
using locally selected methods. Slightly over half (55%) of the 
states reporting workskill attainment as a performance measure were 
using locally selected assessment techniques. 

Infoirmation concerning the specific types of assessment instruments 
and procedures for measuring academic and other performance of 
secondary and postsecondary vocational programs is contained in 
Appendix C (Tables 7-13) . State developed assessment techniques 
included statewide testing instruments, competency checklists, 
common curriculum requirements or high school proficiency exams 
used across the state to assess the academic or other performance 
of vocational students. Some of these instruments are used by the 
states in assessing the performance of all secondary students in 
the state, while others are intended for use with vocational 
students only. 

CONCLUSIONS 

Based on the findings presented in the previous section, a 
numbosr of conclusions have been developed. These conclusions are 
presented in this section. 

In nearly every state, systems of core standards and measures 
of performance for secondary and postsecondary education had been 
developed and implemented in line with the requirements of Section 
115 of the Carl D. Perkins Vocational and Applied Technology 
Education Act of 1990. Only two states had not yet had their 
system approved and were unable to provide the researchers with 
their approved measures for secondary vocational education. At the 
postsecondary level, information was not available from 2 states. 



Table 5 

Assessment Techniques for Basic and Advanced Academic Performance 



Assessment Technique 



States Using 



Secondary 



Post- 
secondarv 





N 


1 


N 




Reading 










State Developed 


23 


54 


3 


12 


Locally Selected/Developed 


12 


28 


15 


57 


Nationally Recognized 


15 


35 


9 


35 


Other 


15 


35 


12 


46 


Language 










State Developed 


23 


58 


3 


10 


Locally Selected/Developed 


13 


33 


14 


48 


Nationally Recognized 


11 


28 


6 


20 


Other 


12 


30 


15 


52 


Math 










State Developed 

Locally Selected/ Developed 


25 


54 


3 


10 


14 


30 


15 


50 


Nationally Recognized 


13 


28 


7 


23 


Other 


14 


30 


14 


47 


Science 










State Developed 


14 


74 


0 


0 


Locally Selected/Developed 


10 


53 


7 


88 


Nationally Recognized 


3 


15 


0 


0 


Other 


7 


37 


4 


50 


Other 










State Developed 


7 


29 


2 


7 


Locally Selected/Developed 


4 


17 


8 


30 


Nationally Recognized 


0 


0 


0 


0 


Other 


3 


13 


9 


33 



Note: Percent of the total number of states reporting a 
performance measure in this category. 
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Table 6 

Assessment Techniques for Competency and Workskill Attainment 



States Using 



Assessment Technique 



Secondary 
N %^ 



Post- 
secondary 



N 



Competency Attainment 



State Developed 


7 


29 


2 


11 


Local Selected/Developed 


16 


67 


12 


67 


Nationally Recognized 


5 


21 


5 


28 


Occupational Certification 
or Licensure 


2 


8 


0 


0 


Other 


7 


29 


7 


39 


Workskill Attainment 










State Developed 


11 


28 


4 


12 


Local Selected/Developed 


27 


69 


19 


55 


Nationally Recognized 


2 


51 


1 


3 


Occupational Certification 
or Licensure 


5 


13 


7 


21 


Other 


8 


20 


8 


24 


Note: ^Percent of the total number 


of states 


reporting 


a 



performance measure in this category. 
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state systems tended to have more measures approved for 
secondary vocational education than they did for postsecondary 
vocational education programs. The average number of performance 
standards that had been approved for secondary vocational education 
programs was 10. An average of 8 performance measures had been 
approved for postsecondary vocational education programs. 

It appeared that performance measures listed in the Carl D. 
Perkins Vocational and Applied Technology Education Act of 1990 
were generally accepted by the states. Of the measures of 
performance included in the Carl D. Perkins Vocational and Applied 
Technology Education Act of 1990, only competency attainment had 
been approved by less than 50% of the states. 

States have taken the requirement that measures of academic 
skills be included in their systems seriously. Academic skill 
measures were approved more often for secondary vocational 
education programs than they were for postsecondary vocational 
education programs. Mathematics and reading were the most 
frequently approved basic and advanced academic skill measure for 
secondary vocational education programs, lullowed by language 
measures. Science measures were approved least often as secondary 
vocational education academic skill measures. For postsecondary 
vocational education, mathematics and language were the most 
frequently approved basic and advanced academic measures followed 
by reading measures. Science measures were also approved least 
often at the postsecondary education level. 

States have also responded positively to the requirement that 
at least one or more measure of performance be included in their 
system of core standards and measures of performance. At the 
secondary vocational education level, work skill attainment 
measures were reported as being approved most often. The next most 
frequently approved measures dealt with placement of any type, 
followed by program completion, and high school graduation. For 
postsecondary vocational education, program completion measures 
were approved most often. The second and third most often approved 
measures were work skill attainment and placement of any type, 
respectively. 

About one-half of the measures for both secondary and 
postsecondary vocational education explicitly addressed the extent 
to which they were serving special populations. In some cases, it 
was not obvious as to whether special population measures could be 
obtained from the state's records or not. 

Specific measures related to the gender mix of individuals 
served by vocatior il education were not widely used by the states. 
Approximately one-third of the states had approved measures of 
gender mix for both secondary and postsecondary vocational 
education programs. 




Attempts at standardizing performance standards and measures across 
the nation will prove challenging due to the diverse nature of the 
approved measures and standards currently in use in the states. 
Also, there is great variability among the types of assessment 
techniques being used. Finally, the states have left much of the 
authority for assessing performance to the local agency, 

RECOMMENDATIONS 

The requirement that states develop a system of standards and 
measures of performance for secondary and postsecondary programs is 
new in vocational education legislation. This initial experience 
should be monitored in order to see how future policy initiatives 
related to these measures and standards might be improved. The 
following specific recommendations are offered: 

1. Information should be collected regarding the rationale 
states used in selecting their standards and measures. 

2. The strengths and weaknesses of the various measures of 
performance should be assessed in order to determine 
their relevancy for future use. Additionally, the 
validity and reliability of these measures should be 
established. 

3. States should critically review their approved system of 
standards and measures of performance in order to 
identify the major facilitators and barriers they have 
encountered in its development and implementation. 
Information also should be collected regarding how states 
offered incentives and made adjustments to encourage 
service to targeted populations, 

4 . Efforts should be made to determine how the state 
approved measures and standards compare with business and 
industry standards. 

5. Further research on the state systems of measures and 
standards should be conducted. Specifically, additional 
analysis of the level and type of standards employed by 
the states would be useful. 

Vocational education has been concerned with evaluation for 
many years. At the national level, vocational education 
legislation has included emphasis on evaluation since the passage 
of the Vocational Education Act of 1963. The inclusion of 
requirements for a state system of performance measures and 
standards in the Carl D. Perkins Vocational and Applied Technology 
Education Act of 1990 continued this emphasis. This study examined 
those systems in early 1993. These performance measures and 
standards should continue to be monitored as they are further 
refined and developed. This information is needed to provide 
information for impioving vocational education programs and guide 
future policy initiatives. 
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ERIC 



November 25, 1992 



1-^ 

Dear 2 : 

The implementation of the Carl D. Perkins Act of 1990 has required states to change how their students are 
being assessed and programs are being evaluated. The statewide system of performance measures and 
standards was to be established by September, 1992. Many states also have revised their procedures for 
assessing program quality. Information on these recently approved standards and measures and the criteria 
used to assess program quality have not been summarized on a national basis. 

We are gathering this information from each state. We need your help in providing the following: 

• A list of the performance standards and measurer for secondary and postsecondary vocational 
education that have been approved by your state board (Section 115). 

• The procedures you will use to make local modiOcations based on economic, geographic, or 
demographic factors, or the characteristics of the population to be served (Section 115). 

• Manuals, procedures, and criteria used to assess local program quality (Section 116). 

• The name of the individual(s) responsible for secondary and/or postsecondary vocational 
education evaluation in each state. A one-page form has been included to provide us with 
this information. 

This information would be especially valuable in developing high quality vocational education programs and 
contributing to the study of vocational education's effectiveness. It would also be important in preparing new 
policies and legislation for vocational education. 

We need to receive this information by December 1, 1992. Thank you for your assistance in providing us with 
this information. We will provide you with a summary of the information we collect. 

Sincerely, 

N. L. McCaslin 
Associate Professor 



William S. Headley 
Graduate Research Associate 
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March 11, 1993 
1^ 

Dear 2" 

Thank you for sending us the information on the statewide system of 
measures and standards required by The Carl Perkins Act of 1990. 
In order to ensure the accuracy of the information we will be 
reporting, we are requesting that you review our findings for your 
state . 

Please check the summary sheet for measures adopted for use in 
secondary and postsecondary programs. Feel free to make any 
necessary additions, deletions, or corrections. If the summary is 
correct, simply write "OK" on the summary sheet and return it to 
us. The enclosed instruction sheet will provide details to assist 
in the review of these materials. We would appreciate any comments 
concerning this summary. A stamped, addressed envelope is provided 
for your use in returning the summary forms. 

The final summary of the findings for all the states will be 
published and made available to you as soon as we have received the 
returned and corrected summary sheets. Therefore, we need to have 
this information by March 29, 1993. Don't hesitate to call if you 
have any questions. Thank you for your assistance in reviewing 
these materials. 

Sincerely, 



N. L. McCaslin 
Associate Professor 



William S. Headley 
Graduate Research Associate 
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Instruction Sheet 

Read over the attached summary sheet. An "x" has been placed 
under the headings indicating your state has a measure in this 
area. Blank spaces indicate no measure in this area. 

Please cross off any incorrect "x" and add any that should be 
present • 

Some fauates are phasing in measures, the measure is listed on 
the summary sheet even if it is to be phased in at a later 
date. 

The notes below will help in the interpretation of the 
headings . 

a. If your state shows an * in the academic skill area, we 
interpreted your system as using the same set of measures 
for both basic and advanced academic skill. 

b. Determined by program means that each vocational program 
area (e.g. agriculture, marketing, business) in the state 
will determine the specific academic measures to be used. 

c. Social studies includes citizenship, American history, 
etc. 

d. Under Placement, both Related and Any include military 
service and further training or education. 

e. Under Special Populations and Other, Enrollment refers to 
measures of numbers of students in programs or ratios of 
students compared to other groups of students, etc. 

f. Program Features refers to measures of curriculum, 
teacher, or other programmatic measures. 

If the summary is correct, simply write "OK" on the summary 
sheet and return it to us. 

Please return the suitoaary forms in the enclosed stamped, 
addressed envelope. 

Your comments are appreciated. If you need additional 
clarification or desire additional information, call Scot 
Headley at 614-292-6321. 
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Table 7 

Technicfues for Assessing Reading Performance 



States Using 

Post- 

Assessment Technigue Secondary sec ondary 



state Developed 


23 


1 


State high school proficiency exam 


10 




Other state developed assessment 


13 


3 


Local Selected or Developed 


12 


15 


Nationally Recoanized 


15 




Iowa Test of Basic Skills 


3 




Test of Adult Proficiency 


3 




ASSET 




1 


Test of Adult Basic Education 


2 


6 


California Test of Basic Skills 


1 




California Achievement Test 




1 


Metropolitan Achievement Test 


1 




Stanford Achievement Test 


1 


1 


Stanford 8 3R battery 


1 




ACT Work Keys 


1 




Gates-MacGinite Reading Test 


1 




VTECS item banks 


1 




Other 


15 


11 


Course completion 


9 


9 


Portfolios 


3 




Grade advancement 


1 




GPA 




2 


GED criteria 


2 


1 



Table 8 



Techniques for Assessing Language Performance 





States Using 






Post- 


Assessment Technigue 


Secondary 


secondary 


State DeveloTDed 


23 


3 


State high school proficiency exam 


9 




n-hViOT* Qtat<a develoiDed assessment 


14 


3 


T.r^r»;^1 ^<al *ar't<ad OT Developed 


13 


14 


National Iv Recognized 


11 


6 


Iowa Test of Basic Skills 


2 




Test of Adult Proficiency 


1 




ASSET 




1 


Test of Adult Basic Education 


2 


5 


California Test of Basic Skills 


1 




Metropolitan Achievement Test 


1 




stanfoTd Achievement Test 


1 




Stanford 8 3R battery 


1 




ACT Work Keys 


1 




VTEGS item banks 


1 




Other 


12 


15 


GPA 




1 


Course completion 


8 


11 


Portfolios 


3 




Institutional requirements 




1 


GED criteria 


1 


1 


Student opinion survey 




1 
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Table 9 

Techniques for Assessing Math Performance 



States Using 

Post- 



Assessment Technigue Secondary secondary 



State Deyelooed 


2S 


1 


State high school proficiency exam 


9 




State deyeloped assessment 


16 


3 




14 


15 




13 


7 


Iowa Test of Basic Skills 


3 




Test of Adult Proficiency 


2 




ASSET 




1 


Test of Adult Basic Education 


2 


5 


California Test of Basic Skills 


1 




Metropolitan Achieyement Test 


1 




Stanford Achieyement Test 


1 


1 


Stanford 8 3R battery 


1 




ACT Work Keys 


1 




VTECS item banks 


1 




Other 


14 


14 


Course completion 


10 


10 


GED 




1 


Portfolios 


3 




Institutional requirements 




1 


Grade advancement 


1 




CPA 






Student opinion survey 




1 
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Table 10 

Technicmes for Assessing Science Performance 



Assessment Technique 


States 
Secondary 


Usina 

Post— 
secondarv 


State Developed 


14 


Q. 


State high school proficiency exam 


7 
7 




T.or»aT_ selected or developed 


10 


7 


Nationally Recoanized 


1 


a. 


California Test of Basic Skills 
Stanford Achievement Test 
Iowa Test of Basic Skills 


1 
1 
1 




Other 


7 


4 


Course completion 
GED criteria 
Portfolios 

Institutional requirements 


6 
1 


2 
1 

1 


Table 11 

Techniques for Assessing Other Academic 


Performance 




Assessment Technique 


States Usina 

Post- 

Secondarv secondary 


State Developed 


7 


1 


State high school proficiency exam 
State developed assessment 


5 
2 


2 


Local Developed or Selected 


4 


8 


Other 


3 


9 


Course completion 
GED criteria 
CPA 


3 


5 
1 
3 
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Table 12 

Techniques for Assessing Competency Attainment 





States 


Usina 






Post- 


Assessment Technique 


Secondary 


secondary 


State Developed 


7 


2 


Local Developed or Selected 


16 


12 


Nationally Recoanized 


5 


5 


Jobs for American Graduates Test 


1 


1 


Workplace Readiness Assessment 


2 


2 


Work Keys ACT 


1 


1 


Youthwork Instrument 


1 


1 


Occupational Licensure or Certification 


2 


0 


Other 


7 


7 


Completion of vocational coursework 


6 


6 


Portfolios 


1 


1 
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Table 13 

Technicmes for Assessing Workskill Attainment 





States Usina 






Post- 


Assessment Technique 


secondary 


secondarv 


State Developed 


11 


4 


State developed tests 


6 


3 


State developed checklist 


5 


1 


Local Developed or Selected 




AS. 


Nauionaxxv Kecocinizea raeasuies 




1 

M 


• 

VTECs materials 


1 




NOCTI Exams 


1 

JL 




Occupational Licensure and Certification 


5 


7 


Other Measures 


8 


8 


Portfolios 


2 




Degree 




1 


Completion of coursework/program 


5 


3 


GPA in vocational courses 


1 


3 


Student opinion survey 




1 
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